Low-light autofocus technique

ABSTRACT

The present disclosure relates to a low-light autofocus technique. One example embodiment includes a method. The method includes receiving an indication of a low-light condition for a camera system. The method also includes determining an extended exposure time for a low-light autofocus procedure of the camera system. Further, the method includes capturing, by the camera system, an extended frame for the low-light autofocus procedure. The extended frame is captured by die camera system using the determined extended exposure time. In addition, the method includes determining, based on the captured extended frame, an in-focus lens setting for a lens of the camera system.

BACKGROUND

Unless otherwise indicated herein, the materials described in thissection are not prior art to the claims in this application and are notadmitted to be prior art by inclusion in this section.

Cameras are devices used to capture images of a scene. Some cameras(e.g., film cameras) chemically capture an image on film. Other cameras(e.g., digital cameras) electrically capture image data (e.g., using acharge-coupled device (CCD) or complementary metal-oxide-semiconductor(CMOS) sensors). In order to most accurately capture the scene, a cameramay be focused on one or more subjects in the scene. There are multipleways to focus a camera. For example, a lens of the camera can be movedrelative to an image sensor of the camera to adjust the focus of thecamera (e.g., to bring one or more subjects in focus). Similarly, animage sensor of the camera can be moved relative to the lens of thecamera to adjust the focus of the camera.

Adjusting the focus of a camera can be performed manually (e.g., by aphotographer). Alternatively, an autofocus procedure can be performed toadjust the focus of a camera prior to capturing an image (e.g., apayload image). Autofocus procedures may use one or more images (eithercaptured by the primary image sensor of the camera or one or moreauxiliary sensors in the camera) to determine an appropriate focussetting for the camera. Then, based on the determined focus setting, thecamera adjusts to meet that focus setting. For example, a motor mayadjust the relative position of the lens and/or the image sensor to meetthe determined focus setting.

There are two traditional types of autofocus procedures, activeautofocus procedures and passive autofocus procedures.

In active autofocus procedures, a rangefinder (e.g., a laserrangefinder, a radar device, or a sonar device) is used to determine adistance to one or more objects within a scene. Then, based on thedetermined distance, a focus setting is determined and the camera isadjusted to meet the determined focus setting.

There are two primary species of passive autofocus procedures,phase-detection autofocus and contrast-detection autofocus.

In phase-detection autofocus, incoming light from the scene is divided(e.g., by a beamsplitter) such that light from the scene entering oneside of the lens of the camera is physically separated on an imagesensor (e.g., the primary image sensor of the camera or an auxiliaryimage sensor) from light from the scene entering the opposite side ofthe lens. Based on the camera characteristics (e.g., lens size, lensfocal length, and lens location relative to the image sensor) and thelight intensity distribution across the various locations on the imagesensor, a focus setting can be determined. Like with active autofocusprocedures, the camera can be adjusted to meet the determined focussetting.

In contrast-detection autofocus, a series of frames are captured by thecamera at a corresponding series of different focus settings. Thecontrast between high intensity and low intensity is then determined foreach of the captured frames. Based on the determined contrasts, a focussetting is determined (e.g., based on the frame with the highestcontrast and/or based on a regression analysis using the contrasts ofthe captured frames). Similar to the active autofocus procedures andphase-detection autofocus, the camera can be adjusted to meet thedetermined focus setting.

Unlike active autofocus procedures, passive autofocus procedures (e.g.,phase-detection autofocus and contrast-detection autofocus) do not useadditional rangefinding equipment. Hence, passive autofocus proceduresmay be employed in camera systems to save on cost (e.g., in a mobilephone or a digital single-lens reflex (DSLR) camera). However, passiveautofocus procedures may be less successful in low-light conditions(e.g., because insufficient contrast is generated between frames for usein contrast-detection autofocus or because there are insufficient brightobjects within a scene to compare when using phase-detection autofocus).

SUMMARY

The specification and drawings disclose embodiments that relate tolow-light autofocus techniques. Performing passive autofocus techniquesin low-light conditions has traditionally been challenging. Exampleembodiments described herein attempt to improve autofocus in low-lightconditions by extending the exposure time for autofocus frames. Theextended exposure time may be determined by evaluating a maximum amountof motion blur that can be tolerated within a frame that is used forautofocus. Based on the amount of motion blur that can be tolerated, anexposure time can be determined. Then, one or more frames can becaptured using the determined exposure time and those frames can be usedfor autofocus. Because these frames may have longer exposure times thantraditional frames, the resulting autofocus may be enhanced. Further,this enhancement does not use additional optics or sensors (unlikeactive autofocus techniques).

In a first aspect, the disclosure describes a method. The methodincludes receiving an indication of a low-light condition for a camerasystem. The method also includes determining an extended exposure timefor a low-light autofocus procedure of the camera system. In addition,the method includes capturing, by the camera system, an extended framefor the low-light autofocus procedure. The extended frame is captured bythe camera system using the determined extended exposure time. Further,the method includes determining, based on the captured extended frame,an in-focus lens setting for a lens of the camera system.

As noted above, an extended frame is a frame captured based upon anextended exposure time. The extended exposure time may be an exposuretime that is longer than an exposure time of a traditional frame. Forexample, the extended exposure time may be longer than the exposure timeset for one or more payload images after the low-light autofocustechnique is performed. In another example, the extended exposure timemay be longer than the exposure time for capturing a frame for anautofocus procedure under normal light conditions. Normal lightconditions may be determined as being above a low-light intensitythreshold as described below.

The method may include the following optional features. Determining anextended exposure time for a low-light autofocus procedure of the camerasystem may be based on a motion-blur tolerance and the low-lightcondition. The method may further include adjusting the lens based onthe determined in-focus lens setting. The method may further includedetermining a confidence level for the in-focus lens setting; performinga comparison of the confidence level to a confidence threshold; andadjusting the lens based on the comparison. Adjusting the lens based onthe comparison may include adjusting the lens to match the in-focus lenssetting in response to the confidence level being greater than or equalto the confidence threshold. Adjusting the lens based on the comparisonmay include adjusting the lens to a default lens setting in response tothe confidence level being less than the confidence threshold. Thedefault lens setting may provide a maximum focal length for the lens.The default lens setting may provide a focal length for the lens thatcorresponds to a hyperfocal distance of the camera system. The methodmay further include capturing, by the camera system, a plurality ofextended frames for the low-light autofocus procedure, whereindetermining the in-focus lens setting for the lens of the camera systemis based upon the plurality of captured extended frames. The method mayfurther include capturing, by the camera system using the adjusted lens,a second extended frame for the low-light autofocus procedure; anddetermining, a second in-focus lens setting for the lens of the camerasystem based upon the second extended frame. The method may furtherinclude adjusting the lens based upon at least of one the following: thesecond in-focus lens setting, a confidence level associated with thesecond in-focus lens setting and a confidence level associated with thein-focus lens setting determined prior to the second in-focus lenssetting. At least one of the extended frames may be captured by thecamera system using an exposure time different than the determinedextended exposure time. The method may further include capturing, by thecamera system, a plurality of additional frames; wherein each of theadditional frames is captured by the camera system using a secondaryexposure time, and wherein the secondary exposure time is shorter thanthe determined extended exposure time; aligning the additional framessuch that similar objects in the additional frames are in similar pixellocations within each of the additional frames; and forming a compositeimage based on the additional frames. The method may further includeenhancing the composite image using a fast Fourier color constancyalgorithm; determining an optimized tone mapping for the compositeimage; and modifying the composite image according to the optimized tonemapping. The method may further include detecting the low-lightcondition for the camera system. Detecting the low-light condition forthe camera system may include comparing an ambient light intensity to athreshold low-light intensity. The threshold low-light intensity may be1.0 lux. The method may further include adjusting the lens to apre-autofocus setting, wherein the pre-autofocus setting includes amiddle focal position for the lens. The camera system may be a componentof a mobile device. The method may further include receiving, via a userinterface of the mobile device, an indication that a still image orvideo image is to be captured using the mobile device. The method mayfurther include displaying, on the user interface of the mobile device,an indication to a user to hold the mobile device still. Determining theextended exposure time may include: determining the extended exposuretime based on a total time over which the autofocus procedure is to beperformed and a number of frames to be captured during the autofocusprocedure. The camera system may be part of a mobile device or a digitalsingle-lens reflex (DSLR) camera, and the total time may be based on amode of operation of the camera system. Determining, based on thecaptured extended frame, the in-focus lens setting for the lens of thecamera system may include applying a phase-detection autofocusalgorithm. The method may further include capturing a plurality offrames, wherein the plurality of captured frames includes afirst-captured frame and one or more subsequently captured framescaptured after the first-captured frame; performing facial recognitionon the first-captured frame to identify a region of interest in thefirst-captured frame and identifying corresponding regions of interestin the one or more subsequently captured frames, wherein applying thephase-detection autofocus algorithm includes using the phase-detectionautofocus algorithm to determine the in-focus lens setting based on thecorresponding regions of interest in the subsequently captured frames.Determining the extended exposure time may further include determiningthe extended exposure time based on a tunable duration that representsan acceptable amount of time to dedicate to the autofocus procedure. Themethod may further include capturing, by the camera system and prior todetermining the extended exposure time, a plurality of preview frames,wherein the motion-blur tolerance is based on a center-weighted averageof motion across the plurality of preview frames. The method may furtherinclude determining the motion-blur tolerance based on a phase-detectionautofocus algorithm.

In a second aspect, the disclosure describes a non-transitory,computer-readable medium having instructions stored therein. Theinstructions, when executed by a processor, perform a method. The methodincludes receiving an indication of a low-light condition for a camerasystem. The method also includes determining an extended exposure timefor a low-light autofocus procedure of the camera system. In addition,the method includes causing the camera system to capture an extendedframe for the low-light autofocus procedure. The extended frame iscaptured by the camera system using the determined extended exposuretime. Further, the method includes determining, based on the capturedextended frame, an in-focus lens setting for a lens of the camerasystem.

In a third aspect, the disclosure describes a mobile device. The mobiledevice includes a camera system. The camera system includes an imagesensor. The camera system also includes a lens configured to modifylight from an environment surrounding the mobile device prior to thelight being detected by the image sensor. Further, the mobile deviceincludes a controller. The controller is configured to receive anindication of a low-light condition for the camera system. Thecontroller is also configured to determine an extended exposure time fora low-light autofocus procedure of the camera system. In addition, thecontroller is configured to cause the camera system to capture anextended frame for the low-light autofocus procedure. The extended frameis captured by the camera system using the determined extended exposuretime. Further, the controller is configured to determine, based on theplurality of captured frames, an in-focus lens setting for the lens.

In an additional aspect, the disclosure describes a system. The systemincludes a means-for receiving a low-light condition for a camerasystem. The system also includes a means-for determining an extendedexposure time for a low-light autofocus procedure of the camera system.Additionally, the system includes a means-for capturing, by the camerasystem, an extended frame for the low-light autofocus procedure. Theextended frame is captured by the camera system using the determinedextended exposure time. Further, the system includes a means-fordetermining, based on the captured extended frame, an in-focus lenssetting for a lens of the camera system.

It will be appreciated that features described above in the context ofone aspect may be combined with features described in the context ofanother aspect.

The foregoing summary is illustrative only and is not intended to be inany way limiting. In addition to the illustrative aspects, embodiments,and features described above, further aspects, embodiments, and featureswill become apparent by reference to the figures and the followingdetailed description.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is an illustration of front, right-side, and rear views of adigital camera device, according to example embodiments.

FIG. 2 is an illustration of a block diagram of a computer device withimage capture capability, according to example embodiments.

FIG. 3A is an illustration of a phase-detection autofocus technique,according to example embodiments.

FIG. 3B is an illustration of a phase-detection autofocus technique,according to example embodiments.

FIG. 3C is an illustration of a phase-detection autofocus technique,according to example embodiments.

FIG. 3D is an illustration of a phase-detection autofocus technique,according to example embodiments.

FIG. 4 is an illustration of a low-light condition in a preview frame,according to example embodiments.

FIG. 5A is an illustration of a frame-capture timeline, according toexample embodiments.

FIG. 5B is an illustration of a frame-capture timeline, according toexample embodiments.

FIG. 5C is an illustration of a frame-capture timeline, according toexample embodiments.

FIG. 5D is an illustration of a frame-capture timeline, according toexample embodiments.

FIG. 6A is an illustration of a portion of a lens-adjustment procedure,according to example embodiments.

FIG. 6B is an illustration of a portion of a lens-adjustment procedure,according to example embodiments.

FIG. 6C is an illustration of a portion of a lens-adjustment procedure,according to example embodiments.

FIG. 7 is a flowchart illustrating a method, according to exampleembodiments.

DETAILED DESCRIPTION

Example methods and systems are described herein. Any example embodimentor feature described herein is not necessarily to be construed aspreferred or advantageous over other embodiments or features. Theexample embodiments described herein are not meant to be limiting. Itwill be readily understood that certain aspects of the disclosed systemsand methods can be arranged and combined in a wide variety of differentconfigurations, all of which are contemplated herein.

Furthermore, the particular arrangements shown in the figures should notbe viewed as limiting. It should be understood that other embodimentsmight include more or less of each element shown in a given figure. Inaddition, some of the illustrated elements may be combined or omitted.Similarly, an example embodiment may include elements that are notillustrated in the figures.

Depending on context, a “camera” may refer to an individual imagecapture component, or a device that contains one or more image capturecomponents. In general, image capture components may include anaperture, lens, recording surface, and shutter, as described below.

The terms “image” and “payload image” may be used herein to describe theultimate image of the scene that is recorded and can be later viewed bythe user of the camera. The term “frame,” on the other hand, may be usedherein to represent temporarily stored depictions of scenes that aredisplayed for preview purposes or are captured and analyzed to determineone or more qualities of a scene prior to capturing a payload image(e.g., to determine what types of subjects are in a given scene, regionsof interest within a given scene, appropriate exposure times, ambientlight intensity, motion-blur tolerance, etc.).

I. OVERVIEW

Example embodiments relate to low-light autofocus techniques. Forexample, some embodiments relate to a method for performing autofocus inlow-light conditions. As described above, performing autofocus inlow-light has traditionally been difficult. Embodiments described hereinseek to improve performance of autofocus in low-light conditions toenhance images captured in low-light conditions.

One example method may be performed using a camera system (e.g., acamera system that is a component of a mobile device, such as a mobilephone, or a DSLR camera). The method may be initiated when it isdetermined that a low-light condition is present. Such a determinationmay be based on, for example, one or more preview frames captured by thecamera system (e.g., and displayed on a display of a mobile device), oneor more frames previously captured by the camera system, a measurementfrom an auxiliary ambient light sensor (e.g., by comparing the ambientlight intensity to an intensity threshold), or a selection by a user ofthe camera system.

After identifying the low-light condition, an exposure time may bedetermined for the low-light autofocus procedure. The exposure time maybe based on a variety of factors. For example, the exposure time may bebased on the low-light condition itself (e.g., the ambient lightintensity), a motion-blur tolerance (e.g., which itself may be based onthe subjects within the scene that is ultimately to be photographedand/or whether the camera system is in a handheld mode or a mountedmode), and/or a maximum amount of time a user is willing to wait for thelow-light autofocus procedure to be performed. The motion-blur tolerancefor performing autofocus may be considerably higher than a motion-blurtolerance for capturing payload images that are ultimately to be viewedby humans. In other words, objects may appear overly and undesirablyblurry in a frame that is acceptable for performing autofocus. Becausethe motion-blur tolerance may be higher for autofocus techniques, thedetermined exposure time for use in the autofocus technique may belonger than the exposure time used to capture traditional images. Byselecting a longer exposure time, additional light can be gathered bythe image sensor through the lens of the camera system. This additionallight can improve the quality of the resulting autofocus algorithmapplied to the captured frame. For example, phase-detection autofocuscan have improved results when using an increased exposure time.

After determining the exposure time, a plurality of frames may becaptured by the camera system. At least one frame of the plurality ofcaptured frames may be captured using the determined exposure time. Thecaptured frames may be used (e.g., by a computing device associated withthe camera system) to determine an in-focus lens setting for the camerasystem. For example, the in-focus lens setting may include the locationof a lens relative to an image sensor such that the subjects in thescene are in focus in payload images. In addition to determining thein-focus lens setting, a confidence level for that in-focus lens settingmay be determined. The confidence level may be determined based on theautofocus algorithm being employed. For example, if a phase-detectionautofocus algorithm is being used to determine an in-focus lens settingbased on the plurality of captured frames, the phase-detection autofocusalgorithm may output a confidence level based on the plurality ofcaptured frames, the subjects in the scene within the captured frames,the determined exposure time, the number of captured frames, and/or thelight intensity within the captured frames.

Based on the in-focus lens setting and the confidence level, the camerasystem may be adjusted. For example, the confidence level may becompared to a confidence threshold. If the confidence level meets orexceeds the confidence threshold, the camera system may adjust tosatisfy the determined in-focus lens setting (e.g., the lens may bemoved relative to the image sensor such that the lens is in thedetermined in-focus position). If the confidence level does not meet orexceed the confidence threshold, though, the camera system may adjust tosatisfy a default focus setting. For example, the lens may be movedrelative to the image sensor such that the lens is at a midpoint of allpossible lens positions, such that the lens has a maximum focal length,or such that the lens corresponds to a hyperfocal distance of the camerasystem.

The autofocus techniques described herein can be used in low-lightconditions, as described above. As such, the autofocus techniquesdescribed herein may be used outdoors at night or in dimly lit rooms. Insome embodiments, the autofocus techniques can be used in extremelow-light conditions, for example, as may occur in astrophotography.

II. EXAMPLE SYSTEMS

The following description and accompanying drawings will elucidatefeatures of various example embodiments. The embodiments provided are byway of example, and are not intended to be limiting. As such, thedimensions of the drawings are not necessarily to scale.

As image capture devices, such as cameras, become more popular, they maybe employed as standalone hardware devices or integrated into variousother types of devices. For instance, still and video cameras are nowregularly included in wireless computing devices (e.g., mobile devices,such as mobile phones), tablet computers, laptop computers, video gameinterfaces, home automation devices, and even automobiles and othertypes of vehicles.

The physical components of a camera may include one or more aperturesthrough which light enters, one or more recording surfaces for capturingthe images represented by the light, and lenses positioned in front ofeach aperture to focus at least part of the image on the recordingsurface(s). The apertures may be fixed size or adjustable. In an analogcamera, the recording surface may be photographic film. In a digitalcamera, the recording surface may include an electronic image sensor(e.g., a charge coupled device (CCD) or a complementarymetal-oxide-semiconductor (CMOS) sensor) to transfer and/or storecaptured images in a data storage unit (e.g., memory).

One or more shutters may be coupled to or nearby the lenses or therecording surfaces. Each shutter may either be in a closed position, inwhich it blocks light from reaching the recording surface, or an openposition, in which light is allowed to reach to recording surface. Theposition of each shutter may be controlled by a shutter button. Forinstance, a shutter may be in the closed position by default. When theshutter button is triggered (e.g., pressed), the shutter may change fromthe closed position to the open position for a period of time, known asthe shutter cycle. During the shutter cycle, an image may be captured onthe recording surface. At the end of the shutter cycle, the shutter maychange back to the closed position.

Alternatively, the shuttering process may be electronic. For example,before an electronic shutter of a CCD image sensor is “opened,” thesensor may be reset to remove any residual signal in its photodiodes.While the electronic shutter remains open, the photodiodes mayaccumulate charge. When or after the shutter closes, these charges maybe transferred to longer-term data storage. Combinations of mechanicaland electronic shuttering may also be possible.

Regardless of type, a shutter may be activated and/or controlled bysomething other than a shutter button. For instance, the shutter may beactivated by a softkey, a timer, or some other trigger. Herein, the term“image capture” may refer to any mechanical and/or electronic shutteringprocess that results in one or more images being recorded, regardless ofhow the shuttering process is triggered or controlled.

The exposure of a captured image may be determined by a combination ofthe size of the aperture, the brightness of the light entering theaperture, and the length of the shutter cycle (also referred to as theshutter length, the exposure length, or the exposure time).Additionally, a digital and/or analog gain (e.g., based on an ISOsetting) may be applied to the image, thereby influencing the exposure.In some embodiments, the term “exposure length,” “exposure time,” or“exposure time interval” may refer to the shutter length multiplied bythe gain for a particular aperture size. Thus, these terms may be usedsomewhat interchangeably, and should be interpreted as possibly being ashutter length, an exposure time, and/or any other metric that controlsthe amount of signal response that results from light reaching therecording surface.

In some implementations or modes of operation, a camera may capture oneor more still images each time image capture is triggered. In otherimplementations or modes of operation, a camera may capture a videoimage by continuously capturing images at a particular rate (e.g., 24frames per second) as long as image capture remains triggered (e.g.,while the shutter button is held down). Some cameras, when operating ina mode to capture a still image, may open the shutter when the cameradevice or application is activated, and the shutter may remain in thisposition until the camera device or application is deactivated. Whilethe shutter is open, the camera device or application may capture anddisplay a representation of a scene on a viewfinder (sometimes referredto as displaying a “preview frame”). When image capture is triggered,one or more distinct payload images of the current scene may becaptured.

Cameras, including digital and analog cameras, may include software tocontrol one or more camera functions and/or settings, such as aperturesize, exposure time, gain, and so on. Additionally, some cameras mayinclude software that digitally processes images during or after imagecapture. While the description above refers to cameras in general, itmay be particularly relevant to digital cameras.

As noted previously, digital cameras may be standalone devices (e.g., aDSLR camera) or integrated with other devices. As an example, FIG. 1illustrates the form factor of a digital camera device 100. Digitalcamera device 100 may be, for example, a mobile device (e.g., a mobilephone), a tablet computer, or a wearable computing device. However,other embodiments are possible. Digital camera device 100 may includevarious elements, such as a body 102, a front-facing camera 104, amulti-element display 106, a shutter button 108, and other buttons 110.Digital camera device 100 could further include a rear-facing camera112. Front-facing camera 104 may be positioned on a side of body 102typically facing a user while in operation, or on the same side asmulti-element display 106. Rear-facing camera 112 may be positioned on aside of body 102 opposite front-facing camera 104. Referring to thecameras as front-facing and rear-facing is arbitrary, and digital cameradevice 100 may include multiple cameras positioned on various sides ofbody 102.

Multi-element display 106 could represent a cathode ray tube (CRT)display, a light-emitting diode (LED) display, a liquid crystal display(LCD), a plasma display, or any other type of display known in the art.In some embodiments, multi-element display 106 may display a digitalrepresentation of the current image being captured by front-facingcamera 104 and/or rear-facing camera 112, or an image that could becaptured or was recently captured by either or both of these cameras.Tus, multi-element display 106 may serve as a viewfinder for eithercamera. Multi-element display 106 may also support touchscreen and/orpresence-sensitive functions that may be able to adjust the settingsand/or configuration of any aspect of digital camera device 100.

Front-facing camera 104 may include an image sensor and associatedoptical elements such as lenses. Front-facing camera 104 may offer zoomcapabilities or could have a fixed focal length. In other embodiments,interchangeable lenses could be used with front-facing camera 104.Front-facing camera 104 may have a variable mechanical aperture and amechanical and/or electronic shutter. Front-facing camera 104 also couldbe configured to capture still images, video images, or both. Further,front-facing camera 104 could represent a monoscopic, stereoscopic, ormultiscopic camera. Rear-facing camera 112 may be similarly ordifferently arranged. Additionally, front-facing camera 104, rear-facingcamera 112, or both, may be an array of one or more cameras.

Either or both of front facing camera 104 and rear-facing camera 112 mayinclude or be associated with an illumination component that provides alight field to illuminate a target object. For instance, an illuminationcomponent could provide flash or constant illumination of the targetobject (e.g., using one or more LEDs). An illumination component couldalso be configured to provide a light field that includes one or more ofstructured light, polarized light, and light with specific spectralcontent. Other types of light fields known and used to recoverthree-dimensional (3D) models from an object are possible within thecontext of the embodiments herein.

Either or both of front facing camera 104 and rear-facing camera 112 mayinclude or be associated with an ambient light sensor that maycontinuously or from time to time determine the ambient brightness of ascene that the camera can capture. In some devices, the ambient lightsensor can be used to adjust the display brightness of a screenassociated with the camera (e.g., a viewfinder). When the determinedambient brightness is high, the brightness level of the screen may beincreased to make the screen easier to view. When the determined ambientbrightness is low, the brightness level of the screen may be decreased,also to make the screen easier to view as well as to potentially savepower. Additionally, the ambient light sensor's input may be used todetermine an exposure time of an associated camera, or to help in thisdetermination.

Digital camera device 100 could be configured to use multi-elementdisplay 106 and either front-facing camera 104 or rear-facing camera 112to capture images of a target object (i.e., a subject within a scene).The captured images could be a plurality of still images or a videoimage (e.g., a series of still images captured in rapid succession withor without accompanying audio captured by a microphone). The imagecapture could be triggered by activating shutter button 108, pressing asoftkey on multi-element display 106, or by some other mechanism.Depending upon the implementation, the images could be capturedautomatically at a specific time interval, for example, upon pressingshutter button 108, upon appropriate lighting conditions of the targetobject, upon moving digital camera device 100 a predetermined distance,or according to a predetermined capture schedule.

As noted above, the functions of digital camera device 100 (or anothertype of digital camera) may be integrated into a computing device, suchas a wireless computing device, cell phone, tablet computer, laptopcomputer, and so on. For example, a camera controller may be integratedwith the digital camera device 100 to control one or more functions ofthe digital camera device 100. For purposes of example, FIG. 2 is asimplified block diagram showing some of the components of an examplecomputing device 200 that may include camera components 224.

By way of example and without limitation, computing device 200 may be acellular mobile telephone (e.g., a smartphone), a still camera, a videocamera, a fax machine, a computer (such as a desktop, notebook, tablet,or handheld computer), a personal digital assistant (PDA), a homeautomation component, a digital video recorder (DVR), a digitaltelevision, a remote control, a wearable computing device, or some othertype of device equipped with at least some image capture and/or imageprocessing capabilities. It should be understood that computing device200 may represent a physical camera device such as a digital camera, aparticular physical hardware platform on which a camera applicationoperates in software, or other combinations of hardware and softwarethat are configured to carry out camera functions.

As shown in FIG. 2, computing device 200 may include a communicationinterface 202, a user interface 204, a processor 206, data storage 208,and camera components 224, all of which may be communicatively linkedtogether by a system bus, network, or other connection mechanism 210.

Communication interface 202 may allow computing device 200 tocommunicate, using analog or digital modulation, with other devices,access networks, and/or transport networks. Thus, communicationinterface 202 may facilitate circuit-switched and/or packet-switchedcommunication, such as plain old telephone service (POTS) communicationand/or Internet protocol (IP) or other packetized communication. Forinstance, communication interface 202 may include a chipset and antennaarranged for wireless communication with a radio access network or anaccess point. Also, communication interface 202 may take the form of orinclude a wireline interface, such as an Ethernet, Universal Serial Bus(USB), or High-Definition Multimedia Interface (HDMI) port.Communication interface 202 may also take the form of or include awireless interface, such as a Wifi, BLUETOOTH®, global positioningsystem (GPS), or wide-area wireless interface (e.g., WiMAX or 3GPPLong-Term Evolution (LTE)). However, other forms of physical layerinterfaces and other types of standard or proprietary communicationprotocols may be used over communication interface 202. Furthermore,communication interface 202 may include multiple physical communicationinterfaces (e.g., a Wifi interface, a BLUETOOTH® interface, and awide-area wireless interface).

User interface 204 may function to allow computing device 200 tointeract with a human or non-human user, such as to receive input from auser and to provide output to the user. Thus, user interface 204 mayinclude input components such as a keypad, keyboard, touch-sensitive orpresence-sensitive panel, computer mouse, trackball, joystick,microphone, and so on. User interface 204 may also include one or moreoutput components such as a display screen which, for example, may becombined with a presence-sensitive panel. The display screen may bebased on CRT, LCD, and/or LED technologies, or other technologies nowknown or later developed. User interface 204 may also be configured togenerate audible output(s), via a speaker, speaker jack, audio outputport, audio output device, earphones, and/or other similar devices.

In some embodiments, user interface 204 may include a display thatserves as a viewfinder for still camera and/or video camera functionssupported by computing device 200. Additionally, user interface 204 mayinclude one or more buttons, switches, knobs, and/or dials thatfacilitate the configuration and focusing of a camera function and thecapturing of images (e.g., capturing a picture). It may be possible thatsome or all of these buttons, switches, knobs, and/or dials areimplemented by way of a presence-sensitive panel.

Processor 206 may include one or more general purpose processors (e.g.,microprocessors) and/or one or more special purpose processors (e.g.,digital signal processors (DSPs), graphics processing units (GPUs),floating point units (FPUs), network processors, or application-specificintegrated circuits (ASICs)). In some instances, special purposeprocessors may be capable of image processing, image alignment, andmerging images, among other possibilities. Data storage 208 may includeone or more volatile and/or non-volatile storage components, such asmagnetic, optical, flash, or organic storage, and may be integrated inwhole or in part with processor 206. Data storage 208 may includeremovable and/or non-removable components.

Processor 206 may be capable of executing program instructions 218(e.g., compiled or non-compiled program logic and/or machine code)stored in data storage 208 to carry out the various functions describedherein. Therefore, data storage 208 may include a non-transitory,computer-readable medium, having stored thereon program instructionsthat, upon execution by the processor 206, cause computing device 200 tocarry out any of the methods, processes, or operations disclosed in thisspecification and/or the accompanying drawings. For example, the programinstructions 218, when executed by the processor 206, may perform one ormore autofocus techniques described herein. The execution of programinstructions 218 by processor 206 may result in processor 206 using data212.

By way of example, program instructions 218 may include an operatingsystem 222 (e.g., an operating system kernel, device driver(s), and/orother modules) and one or more application programs 220 (e.g., camerafunctions, address book, email, web browsing, social networking, and/orgaming applications) installed on computing device 200. Similarly, data212 may include operating system data 216 and application data 214.Operating system data 216 may be accessible primarily by operatingsystem 222, and application data 214 may be accessible primarily by oneor more of application programs 220. Application data 214 may bearranged in a file system that is visible to or hidden from a user ofcomputing device 200.

Application programs 220 may communicate with operating system 222through one or more application programming interfaces (APIs). TheseAPIs may facilitate, for instance, application programs 220 readingand/or writing application data 214, transmitting or receivinginformation via communication interface 202, receiving and/or displayinginformation on user interface 204, and so on.

It is understood that application programs 220 may sometimes be referredto as “apps” for short. Additionally, application programs 220 may bedownloadable to computing device 200 through one or more onlineapplication stores or application markets. However, application programscan also be installed on computing device 200 in other ways, such as viaa web browser or through a physical interface (e.g., a USB port) oncomputing device 200.

Camera components 224 may include, but are not limited to, an aperture,shutter, recording surface (e.g., photographic film and/or an imagesensor), lens, and/or shutter button. Camera components 224 may becontrolled at least in part by software executed by processor 206.

FIG. 3A is an illustration of a phase-detection autofocus technique.Phase-detection autofocus is a passive autofocus technique that attemptsto determine a proper focus setting of a camera system (e.g., a positionof a lens and/or a position of an image sensor) based on the subjectswithin a scene of a surrounding environment that will ultimately becaptured in a payload image. Phase-detection autofocus functions bysplitting light that enters the camera system (e.g., via a lens 310, asillustrated in FIG. 3A) into two or more portions. Those portions mayare captured and then compared to one another. The two or more portionsare compared to determine relative locations of intensity peaks andvalleys across the respective frames. If the relative locations withinthe frame match, the subject(s) of the scene are in focus. If therelative locations do not match, then the subject(s) of the scene areout of focus. Based on the distance between respective peaks andrespective valleys and the position of optics within the camera system(e.g., the lens 310, image sensor(s), etc.), adjustments can bedetermined that would move the subject(s) into focus. This is detailedfurther below.

In some embodiments, light from the scene that is split forphase-detection autofocus purposes may be split by a prism or amicro-lens array. The two or more frames from the split light may thenbe directed to two or more different image sensors within the camerasystem or two or more portions of the same image sensor within thecamera system. For example, FIG. 3A illustrates a first light signal 312that comes from a left portion of the lens 310 and a second light signal314 that comes from a right portion of the lens 310. As depicted, thefirst light signal 312 and the second light signal 314 may be directedto different regions of an image sensor 320 of the camera system. Theimage sensor 320 of FIG. 3A may be the primary image sensor of thecamera system (e.g., used to capture payload images) or an auxiliaryimage sensor (e.g., dedicated for use in phase-detection autofocus).

A first depiction of the scene 322 that corresponds to the first lightsignal 312 may then be captured by the image sensor 320. Similarly, asecond depiction of the scene 324 that corresponds to the second lightsignal 314 may be captured by the image sensor 320. The first and seconddepictions of the scene 322, 324 are illustrated as protruding from thesurface of the image sensor 320 for illustrative purposes only. It isunderstood that these depictions would, in reality, correspond tochemical or digital recordings of intensity distributions of the firstand second light signals 312, 314 on a surface of the image sensor 320.As illustrated, the first depiction of the scene 322 is separated fromthe second depiction of the scene 324 on the image sensor 320 (and,thus, correspondingly, within a captured frame) by a distance d. Thedistance d has a value of a as illustrated in FIG. 3A. Hence, based onthe distance d not being equal to 0, it could be determined that thesubjects in the scene are out of focus. Further, based on the value aand the configuration of the camera system (e.g., focal length of thelens 310, position of the lens 310 relative to the image sensor 320,etc.), the amount that the subjects in the scene are out of focus and anadjustment (e.g., to the lens 310 position) to bring them in focus canbe determined.

In alternate embodiments, rather than being directed to differentregions of the image sensor 320, the light signals 312, 314 could besplit at the pixel level (e.g., by a micro-lens overlying each pixel ofthe image sensor 320) and directed to different regions of each pixel(e.g., a left light detector of a pixel and a right light detector of apixel). In such embodiments, the intensity peaks and valleys could becompared at a pixel-by-pixel level or at a frame level, where two framesare generated, one frame being a composite of all detections from theleft light detectors of the pixels and a second frame being a compositeof all detections from the right light detectors of the pixels. In otherembodiments, rather than being directed to different regions of theimage sensor 320, the light signals 312, 314 could be split and directedto different image sensors entirely (e.g., one image sensor thatcaptures a frame based on the first light signal 312 and one imagesensor that captures a frame based on the second light signal 314). Aframe captured by the image sensor corresponding to the first lightsignal 312 could then be compared to a frame captured by the imagesensor corresponding to the second light signal 314.

Further, in some embodiments, one or more objects in the scene may be infocus while others remain out of focus. Hence, determining whether thescene is out of focus may include selecting one or more subjects in thescene upon which to make the determination. A region of interest forfocus determination may be selected based on a user (e.g., of a mobiledevice). For example, the user may select an object in a preview framethat the user desires to be in focus (e.g., a building, a person, theface of a person, a car, etc.). Alternatively, an object-identificationalgorithm may be employed to determine what type of objects are in ascene and determine which of the objects should be in focus based on alist ranked by importance (e.g., if a human face is in the scene, thatshould be the object in focus, followed by a dog, followed by abuilding, etc.). In still other embodiments, whether the scene is infocus or out of focus may include identifying whether an object that ismoving within a scene (e.g., as determined by preview frames) is infocus or not. Alternatively, determining an “in focus” camera settingcould include determining the lens setting at which a maximized regionof the frame (e.g., by pixel area) is in focus or a maximized number ofsubjects (e.g., one discrete object, two discrete objects, threediscrete objects, four discrete objects, etc.) within the frame are infocus.

Similar to FIG. 3A, FIG. 3B illustrates a phase-detection autofocustechnique. Like FIG. 3A, the first light signal 312 and the second lightsignal 314 are split and imaged by the image sensor 320. The differencebetween FIG. 3B and FIG. 3A is the separation between the lens 310 andthe image sensor 320. In FIG. 3B, the image sensor 320 is positionedcloser to the lens 310 than in FIG. 3A. Because the image sensor 320 iscloser to the lens 310, the corresponding first and second depictions ofthe scene 322, 324 have switched sides of the captured frame on theimage sensor 320 when compared to FIG. 3A. Like with FIG. 3A, the firstand second depictions of the scene 322, 324 are again separated by adistance d. However, in FIG. 3B, d is equal to −a. Again, because d isnot equal to 0, the scene may be determined to be out of focus given thecurrent settings of the camera system. Further, based on the value −aand the configuration of the camera system (e.g., focal length of thelens 310, position of the lens 310 relative to the image sensor 320,etc.) the amount that the subjects in the scene are out of focus and anadjustment (e.g., to the lens 310 position) to bring them in focus canbe determined.

Similar to FIGS. 3A and 3B. FIG. 3C illustrates a phase-detectionautofocus technique. Like FIGS. 3A and 3B, the first light signal 312and the second light signal 314 are split and imaged by the image sensor320. The difference between FIG. 3C and FIGS. 3A and 3B is theseparation between the lens 310 and the image sensor 320. In FIG. 3C,the image sensor 320 is positioned closer to the lens 310 than in FIG.3A but farther from the lens 310 than in FIG. 3B. Because of theposition of the image sensor 320, the corresponding first and seconddepictions of the scene 322, 324 now coincide with one another (i.e.,overlap) in the captured frame on the image sensor 320. Unlike in FIGS.3A and 3B, the distance d between the first and second depictions of thescene 322, 324 is equal to 0. Because d is equal to 0, the scene may bedetermined to be in focus given the current settings of the camerasystem. Hence, the camera system may be sufficiently in focus to capturea payload image of the scene.

FIG. 3D uses two potential frames captured by image sensors at differentlocations to illustrate a phase-detection autofocus technique. The firstphase-detection frame is illustrated as captured by the image sensor ata first location 360. Further, the first phase-detection frame includesa first instance 362 of a subject and a second instance 364 of the samesubject based on the separated light signals (similar to the firstdepiction of the scene 322 and the second depiction of the scene 324illustrated and described with respect to FIG. 3A). As illustrated, thefirst instance 362 and the second instance 364 are separated in theframe by a distance of d=a (similar to FIG. 3A). If, instead, the framehad been captured at a second location 370 of the image sensor, which isa distance x closer to the lens 310 than the first location 360, thefirst instance 362 and the second instance 364 of the subject mayoverlap (i.e. d would be equal to 0). Hence, if a frame is captured atthe first location 360, the goal would be to ascertain the distance xbased on the first frame, such that the lens 310 can move relative tothe image sensor by a distance x to put the subjects of the scene infocus in order to prepare to capture a payload image.

Based on the location of the lens 310, the characteristics of the lens310 (e.g., focal length), the location of the image sensor, and/or otherqualities of the camera system, properties of the triangle 390illustrated in FIG. 3D are known. For example, the angles θ and ϕ may beknown based on the camera system. Also, the distance d(d=a in FIG. 3D)can be determined based on the frame captured at the first location 360.Using the distance d and the angles θ and ϕ, the distance x can bedetermined trigonometrically. Thus, based on the captured frame and thegeometry of the optics within the camera system, an adjustment thatcould be made (e.g., by moving the lens 310 and/or the image sensorusing a motorized stage) to bring subject(s) of the scene into focuscould be determined.

As demonstrated above, a single frame can be used to determine theamount a scene is out of focus using a phase-detection autofocusalgorithm. In alternate embodiments, however, multiple frames may becaptured. After each frame is captured, a calculation may be made basedon the captured frame to determine the proper lens setting to bring thesubject(s) of the surrounding scene in focus. The first framecapture/calculation may be coarse with each additional framecapture/calculation being slightly more refined until a sufficientin-focus setting has been achieved (e.g., each additional framecapture/calculation may provide convergence toward the actual focusvalue).

While the technique described above is adequate for performing autofocusin many applications, there is a threshold amount of light that shouldbe captured in the frame used to perform the phase-detect autofocus.This may be challenging in low-light situations (e.g., outdoors at nightor in a darkly lit room). As such, techniques described herein can beused to enhance the amount of light that is used for phase-detectionautofocus.

FIG. 4 is an illustration of a low-light condition in a preview frame402, according to example embodiments. The preview frame 402 may displaya captured frame to a user based on the current scene being capturedusing the current camera system settings (e.g., aperture settings,exposure settings, etc.). The techniques described herein may be usedwhen a preview frame appears similar to the preview frame 402 of FIG. 4.For example, if a payload image is to be captured (e.g., as determinedby a user pressing a shutter button such as a physical shutter button ona DSLR or a shutter button on the user interface (UI) of a mobiledevice), an ambient light intensity may be determined (e.g., based on apreview frame, the ambient light intensity incident upon the imagesensor, or the ambient light intensity incident on an auxiliary sensor).This ambient light intensity may be compared to a threshold low-lightintensity to determine if a low-light condition is present. Thethreshold low-light intensity could be 3.0 lux, 2.5 lux, 2.0 lux, 1.5lux, 1.0 lux, 0.5 lux, etc., in various embodiments. If the ambientlight intensity detected is below the threshold low-light intensity, thelow-light autofocus procedures described herein may be performed. If theambient light intensity is greater than or equal to the thresholdlow-light intensity, the low-light autofocus procedures described hereinmay not be performed or may be performed with modifications (e.g.,different exposure times). In some embodiments, an indication that apayload image is to be captured may include a mobile device (e.g., thedigital camera device 100) receiving (e.g., via a user interface) anindication that a still image or video image is to be captured using themobile device.

Alternatively, the low-light autofocus procedures described herein maybe triggered when a user presses a shutter button (e.g., a physicalshutter button on a DSLR or a shutter button on the UI of a mobiledevice) within a low-light mode (e.g., an application within a mobiledevice or tablet computing device) or otherwise selects a low-lightmode. In such embodiments, the user may make a determination when theambient light intensity is low enough to warrant an extended autofocusprocedure and/or may provide an indication that she is willing to waitfor an extended autofocus procedure to be executed.

In still other embodiments, the low-light autofocus procedures describedherein may be triggered when a previous autofocus has been unsuccessful.For example, the preview frame 402 illustrated in FIG. 4 may haveinadequate focus for a payload image, so the low-light autofocusprocedure may be executed. Alternatively, a previously captured payloadimage could be deemed to have inadequate focus based on a previousautofocus procedure. Whether or not the previous autofocus wasunsuccessful could be based on an indication from a user that theprevious autofocus was inadequate. In other embodiments, an autofocusalgorithm (e.g., a phase-detection autofocus algorithm used in previewmode) may provide an indication that the autofocus has failed. Forexample, the autofocus algorithm may provide a confidence value thatindicates the probability that the autofocus was successful, and if thatconfidence value is below a certain threshold, it may be determined thatthe autofocus failed. An indication that the autofocus has failed may beprovided by an API (e.g., an API for a camera module of the mobiledevice).

Upon the extended autofocus procedure engaging, an indication can beprovided to a user (e.g., at the beginning of the extended autofocusprocedure or throughout the duration of the autofocus procedure). Forexample, an indication could be displayed on the multi-element display106 of the digital camera device 100 that says “extended autofocus beingperformed” or “please hold camera still, extended autofocus beingperformed” (i.e., an indication to the user of the camera to hold thecamera still). The indication displayed on the multi-element display 106of the digital camera device 100 could additionally or alternativelyinclude a timer that indicates how much time remains in the extendedautofocus procedure. Such an indication could be displayed on top of ashutter button engaged by the user, for example.

Once it has been determined that an autofocus procedure is to beexecuted, the number of frames and the duration of those frames may bedetermined and then the extended autofocus frames may be captured. Forexample. FIG. 5A illustrates a frame-capture timeline in line with theautofocus procedures described herein. As illustrated, the frame-capturetimeline includes two extended frames 502 for use in an autofocusalgorithm (e.g., a phase-detection autofocus algorithm). The exposuretime for each of these extended frames 502 is indicated by their extentalong the time axis in FIG. 5A. The exposure time for these extendedframes 502 may be greater than an exposure time used for the payloadimage(s) after the autofocus technique is performed. One reason for thisis that the autofocus algorithm (e.g., the phase-detection autofocusalgorithm) may have enhanced motion-blur tolerance relative to a payloadimage. Because the two depictions of the scene captured in the extendedframes 502 (e.g., similar to the first and second depictions of thescene 322, 324 illustrated in FIG. 3B) are captured simultaneously,motion blur does not adversely affect the extended frames 502 when thoseframes are used for phase-detection autofocus rather than generating apayload image.

The exposure time for the extended frames 502 may be determined based ona total time over which the autofocus procedure is to be performed and anumber of frames to be captured during the autofocus procedure. Forexample, a user may indicate that she is accepting of 6.0 seconds toperform the extended autofocus procedure. If two frames are to becaptured, that 6.0 second duration may be split equally across bothframes giving each of the extended frames 502 3.0 seconds of exposuretime. The amount of time a user is willing to dedicate to the extendedautofocus procedure may be tunable (e.g., by the user).

As another example, the camera system may be in a handheld mode (e.g.,if the camera system is a DSLR camera or a mobile phone), so it may bedetermined that 3.0 seconds are to be used to perform the extendedautofocus procedure (e.g., instead of 6.0 seconds). If, again, there aretwo frames, each frame may be 1.5 seconds each. Alternatively, one framemay be 2.0 seconds while the other is 1.0 seconds. However, if thecamera system is in a mounted or a “tripod” mode (e.g., if the camerasystem is a DSLR camera positioned on a tripod), it may be determinedthat 9.0 seconds are to be used to perform the extended autofocusprocedure. If there are three frames to be captured, for example, eachframe may have 3.0 seconds of exposure time. As described herein, themode a camera system is operating in (e.g., handheld vs. mounted) may beused in determining the total time over which the autofocus proceduremay be performed and/or the exposure time for individual frames to becaptured.

In some embodiments, the exposure time for the extended frames 502 mayalso be determined based on the ambient light intensity of theenvironment (i.e., the brightness of the scene). The ambient lightintensity may be determined based on the light intensity incident on theimage sensor of the camera system (e.g., as captured in one or morepreview frames or previous payload images) and/or based on an auxiliarysensor that detects light intensity. The greater the ambient lightintensity, the less exposure time is necessary for adequate autofocusdeterminations to be made based on the captured frames. Hence, inlow-light conditions, the exposure time for the extended frames 502 maybe less than in ultra-low-light conditions.

The exposure time may also be based on a motion-blur tolerance. Themotion-blur tolerance may depend directly on the phase-detectionautofocus algorithm that is to be executed on the extended frames 502once they are captured. Alternatively, the motion-blur tolerance may bedetermined based on the motion of subject(s) in the scene, as describedwith reference to FIG. 5D below.

Other methods of determining the total time for exposure of the extendedframes 502 are also possible and described and contemplated herein.

After calculating the exposure time for the extended frames 502, theextended frames 502 may be captured (e.g., using an API call within thecamera application of a mobile computing device). For example, each ofthe extended frames 502 could be captured using an exposure time of 1.0second each, 1.5 seconds each, 2.0 seconds each, 2.5 seconds each, 3.0seconds each, 3.5 seconds each, 4.0 seconds each, etc. While FIG. 5Aillustrates two extended frames 502, it is understood that other numbersof extended frames 502 could be captured for the purpose of autofocus(e.g., one, three, four, five, six, seven, eight, nine, ten, fifteen,twenty, etc.). Only one extended frame 502 is necessary to performphase-detection autofocus. However, it may be beneficial to useadditional extended frame(s) 502 to improve the resolution of thein-focus lens setting and/or to confirm that the in-focus lens settingdetermined based on the first extended frame 502 was determinedcorrectly. Once captured, the extended frames 502 may be used for anautofocus procedure (e.g., fed into a phase-detection autofocusalgorithm).

As described throughout, phase-detection autofocus procedures may beused to determine the proper lens setting for a camera system to bringsubjects of a frame into focus prior to capturing a payload image.However, it is understood that other autofocus techniques are understoodand could alternatively be used. For example, contrast-detectionautofocus could be used. In embodiments using contrast-detectionautofocus, additional frames may be captured to perform autofocus, witheach frame being captured at a different focus setting. This may includecapturing additional extended frames 502, possibly using differentexposure times. Contrast-detection autofocus may also take longer as thelens setting may change between exposures (which can take time), as wellas because additional extended frames 502 may be captured. Further,contrast-detection autofocus may have lower motion-blur tolerancebecause contrast-detection autofocus is reliant on successively capturedframes, rather than on single/simultaneously captured frames wheremotion blur will be equally prevalent in all depictions of the scene (asin phase-detection autofocus). Hence, due to the lower motion-blurtolerance of a contrast-detection autofocus algorithm, the exposure timefor the extended frames 502 when using contrast-detection autofocus maybe limited. This may correspondingly reduce performance in low-lightconditions. However, there is nothing that prevents contrast-detectionautofocus from being used herein and nothing in this disclosure shouldbe read so as to preclude the use of contrast-detection autofocus orother autofocus techniques besides phase-detection autofocus.

In some embodiments, the exposure time for one of the extended frames502 may not be the same as for the rest of the extended frames 502. Forexample, as illustrated in FIG. 5B, the first of the extended frames 502may have a longer exposure time than the second of the extended frames502. In some embodiments, the second or subsequent extended frames 502may be present to either provide fine (as opposed to coarse) adjustmentsto the focus setting for the camera system or to confirm the focussetting for the camera system determined based on the first of theextended frames 502. As such, the exposure time for the second extendedframe 502 may be shorter than the exposure time for the first extendedframe 502. In other embodiments, the second or subsequent extendedframes 502 may be longer than the first extended frame 502.

In addition, in some embodiments, additional frames may be capturedafter the extended frames 502. For example, as illustrated in FIG. 5C,multiple payload frames 522 may be captured after the extended frames502. Prior to capturing the payload frames 522, there may be a briefresting period (indicated in FIG. 5C by the open space along the timeaxis between the extended frames 502 and the payload frames 522). Theresting period may allow for the lens and/or image sensor to be movedwithin the camera system based on the focus setting determined by theextended frames 502 (i.e., allows the camera system time to readjustsuch that the scene is in focus). This ensures that the payload image isnot captured while the lens and/or the image sensor are still in motion.In some embodiments, the resting period may include requesting anadditional frame using an API, but then not using the requested frameand/or disposing of the requested frame.

The plurality of payload frames 522 may be captured and used to generatea payload image. As illustrated, each of the payload frames 522 may becaptured using a different exposure time than was used for the extendedframes 502 (e.g., a shorter exposure time than was used for the extendedframes 502). In some embodiments, as illustrated in FIG. 5C, the payloadframes 522 may all be captured using the same exposure time as oneanother. In other embodiments, the payload frames 522 may be capturedusing two or more different exposure times. To generate the payloadimage, the payload frames 522 may be aligned such that similar objectsin the payload frames 522 are in similar pixel locations within each ofthe payload frames 522 (e.g., so the subject of the payload frame 522 isin the same position within each of the payload frames 522). Generatingthe payload image may also include forming a composite image based onthe aligned payload frames 522.

In some embodiments, generating the payload image using the payloadframes 522 may include enhancing the composite image using a fastFourier color constancy algorithm. Further, generating the payload imagemay include determining an optimized tone mapping for the compositeimage generated from the payload frames 522 and modifying the compositeimage generated from the payload frames according to the optimized tonemapping. The payload image generated may be a high dynamic range (HDR)image that is enhanced in low-light conditions based on the extendedautofocus procedure described herein.

In some embodiments, the exposure time for the extended frames 502 maybe based on a motion-blur tolerance. The motion-blur tolerance maydepend on the autofocus algorithm being used (e.g., hard-coded into theautofocus algorithm). Alternatively, the motion-blur tolerance may bebased on the subjects in a given scene. For example, as illustrated inFIG. 5D, preview frames 532 could be captured prior to capturing theextended frames 502. A center-weighted average of motion (or othermetric of motion) across the plurality of preview frames 532 may then bedetermined and the motion-blur tolerance may be based on thiscenter-weighted average. The center-weighted average looks at the motionof objects (e.g., bright objects, such as taillights on vehicles) fromone preview frame 532 to the next and more heavily weights motionoccurring in the center of the frames because that motion is more likelyto constitute an important subject within the frames. Based on thiscenter-weighted average, an amount of motion in a scene can be detected.The more motion that is present in a scene, the greater the motion blurfrom frame to frame and, therefore, generally, the shorter theacceptable exposure time.

Additionally or alternatively, the motion of one or more region(s) ofinterest within the preview frames 532 may be tracked to determine themotion-blur tolerance used in determining the exposure time of theextended frames 502. The region(s) of interest within the preview frames532 may be identified by a user (e.g., by a user selecting one or moreregions or objects in the preview frames 532 using a UI) or identifiedby a computing device (e.g., a processor executing a machine-learnedfacial recognition algorithm or a machine-learned object detection andidentification algorithm). The region(s) of interest may be identifiedwithin the first preview frame 532 and then, based on the region(s) ofinterest in the first preview frame 532 and corresponding region(s) ofinterest in subsequent preview frames 532, exposure times for use incapturing the extended frames 502 may be used.

As stated above, the preview frames 532 may be captured prior tocapturing the extended frames 502. The preview frames 532 may be usedfor other purposes in addition to determining exposure time (e.g., basedon motion-blur tolerance and ambient light intensity) for the extendedframes 502. For example, many cameras may include a viewfinder. When thecamera's aperture is open and/or when the camera is otherwise ready tocapture a payload image, preview frames 532 may be displayed in theviewfinder. These preview frames 532 may be refreshed at a particularrate, referred to as the viewfinder refresh rate. In some cases, theviewfinder refresh rate is 15 Hz or 30 Hz, but other rates may be used.These refresh rates define the viewfinder's refresh time interval, whichis the amount of time between refreshes. The refresh time interval isthe inverse of the refresh rate—thus, a refresh rate of 30 Hz has arefresh time interval of 33.33 milliseconds, while a refresh rate of 15Hz has a refresh time interval of 66.66 milliseconds.

A user may utilize the preview frames 532 to aim, focus, or otherwiseadjust the image capture device. In some embodiments, a user may selecta region of interest within one or more of the preview frames 532 basedon what subjects are depicted within the preview frame. For example, ifa person is depicted in the preview frames 532 in front of a landscape,a user may select the entire person or a facial region of the person asa region of interest for subsequent capture in a payload image. In somescenarios, once the user is satisfied by what he or she sees on theviewfinder, the user triggers the image capture device's shutterfunction (e.g., using a shutter button). This may result in a payloadimage being captured (e.g., with higher resolution than the viewfinderframes). The payload image may be stored to memory and/or presented tothe user as the actual photographed image. The user may then share,print, or further manipulate this payload image.

As described above, standard autofocus (e.g., phase-detection autofocus)on preview frames with shorter exposure times than the extended frames502 can be attempted prior to determining that an extended autofocustechnique is to be performed. This may also be illustrated by FIG. 5D. Aseries of preview frames 532 are captured (e.g., using a shorterexposure time than the exposure time for the extended frames 502) priorto capturing the extended frames 502. If the standard autofocus fails onthe preview frames 532, the extended autofocus may be triggered, therebycausing the extended frames 502 to be captured/analyzed. As alsodescribed above, the preview frames 532 can be used to compute acenter-weighted average of motion in order to determine the motion-blurtolerance.

It is understood that the relative durations of the exposure timesillustrated in FIG. 5D are illustrative only and not necessarily toscale. Other exposure times for the preview frames 532, the extendedframes 502, and/or the payload frames 522 may be possible. Additionally,other numbers of preview frames 532, extended frames 502, and/or payloadframes 522 are also possible.

As described above, the lens 310 of a camera system may be movedrelative to the image sensor 320 in response to a determination of anin-focus lens setting using captured extended frames 502. Such amovement is depicted in FIGS. 6A and 6B. It is understood that otherchanges to the lens 310 (besides simply repositioning the lens 310)could be made to match the in-focus lens setting in response to aconfidence level of the determined in-focus setting being greater thanor equal to the confidence threshold. For example, the lens 310 could bewarped and/or deformed so as to change its focal length.

The arrangement illustrated in FIG. 6A may show one position of the lens310 (e.g., a pre-autofocus setting 612, as depicted in FIG. 6B) relativeto the image sensor 320 prior to performing the extended autofocusprocedure (e.g., prior to when the extended frames 502 are captured).Unlike in FIGS. 6A and 6B, in some embodiments, the lens 310 may bepositioned at a middle focal position for the lens 310 (e.g., at amidpoint of all possible positions of the lens 310 within the camerasystem). In this way, the value of d, as illustrated in FIGS. 3A-3C, forexample, is equally as likely to be positive as it is to be negative.Said another way, a priori, a positive adjustment to the lens 310 isjust as likely to bring the scene into focus as a negative adjustment tothe lens 310. This may reduce the amount of adjustment necessary afterthe extended autofocus technique is performed to bring the camera systemin line with the in-focus lens setting. Similarly, the midpoint can leadto enhanced accuracy for autofocus determination (e.g., thephase-detection autofocus algorithm may be more accurate when assessingvalues of d that are closer to 0). It is understood that other locationsof the lens 310 relative to the image sensor 320 (e.g., other than themiddle focal position and the position illustrated in FIG. 6A) may beused for the pre-autofocus setting 612.

In some embodiments, the pre-autofocus setting for the lens 310 may bethe same position prior to each extended autofocus procedure. Forexample, the lens 310 may be moved to the same position relative to theimage sensor 320 prior to each execution of the extended autofocusprocedure. This position may be referred to as a pre-autofocus setting612 (e.g., as depicted in FIG. 6B). Using the same pre-autofocus setting612 prior to each extended autofocus procedure may mean that the resultsof one extended autofocus procedure can be compared against otherextended autofocus procedures. Hence, the results of previouslyperformed extended autofocus techniques can be used to enhanceadditional enhanced autofocus techniques.

As described above, the extended autofocus procedure may result in adetermined in-focus lens setting and a confidence level. The confidencelevel may be output from the phase-detection autofocus algorithm (e.g.,based on the ambient light intensity, the amount of motion in the scene,the objects in the scene, the relative separation between the variousintensity peaks and valleys in the captured frame(s), etc.). Thereafter,the confidence level may be compared to a confidence threshold. Theconfidence threshold may be selectable/tunable by a user, may be storedin a memory associated with the camera system, and/or may be determinedbased on the phase-detection autofocus algorithm used. If the confidencelevel is greater than or equal to the confidence threshold, the lens 310may be moved or otherwise adjusted to satisfy the in-focus lens setting.For example, as illustrated in FIG. 6B, the lens 310 may be moved fromthe pre-autofocus setting 612 to the in-focus lens setting 614. Movingthe lens 310 may include translating the lens 310 within the camerasystem using an electrically controlled stage and/or a motor, forexample.

In cases where the determined confidence level is less than theconfidence threshold, however, the lens 310 may not be moved to thedetermined in-focus lens setting. Instead, as illustrated in FIG. 6C,the lens 310 may be moved from the pre-autofocus setting 612 to adefault setting 616. As with FIG. 6B, this movement may be performedusing an electrically controlled stage and/or a motor. The defaultsetting 616 may correspond to the middle focal position of the lens 310,in some embodiments. Alternatively, the default setting 616 maycorrespond to a maximum focal length for the lens 310. For example, thedefault setting 616 may be set such that a focal distance of infinity isas nearly approximated as possible by the camera system. In still otherembodiments, the default setting 616 may correspond to a hyperfocaldistance of the camera system (i.e., the focus setting that correspondsto a focal distance beyond which all objects in the scene can be broughtinto acceptable focus, where the acceptable focus may be defined by thecircle of confusion diameter limit).

In some embodiments, regardless of the outcome of the comparison betweenthe confidence level and the confidence threshold, the lens 310 may bemoved to a default setting 616. For example, if the ambient lightintensity is below 0.25 lux, 0.5 lux, 0.75 lux, 1.0 lux, etc.,regardless of the computed confidence level and the value of theconfidence threshold, the lens 310 may be moved to the default setting616.

III. EXAMPLE PROCESSES

FIG. 7 is a flowchart illustrating a method 700, according to exampleembodiments. The method 700 may be performed by the digital cameradevice 100 illustrated in FIG. 1 and/or the computing device 200illustrated in FIG. 2, for example.

At block 702, the method 700 may include detecting a low-light conditionfor a camera system. The camera system may correspond to the digitalcamera device 100, in some embodiments.

In some embodiments, block 702 may include comparing an ambient lightintensity to a threshold low-light intensity. For example, the thresholdlow-light intensity may be 1.0 lux.

At block 704, the method 700 may include determining an exposure timefor an autofocus procedure of the camera system. Determining theexposure time may include determining the exposure time based on amotion-blur tolerance and the low-light condition.

In some embodiments, block 704 may include determining the exposure timebased on a total time over which the autofocus procedure is to beperformed and a number of frames to be captured during the autofocusprocedure. For example, the camera system may be part of a mobile deviceor a DSLR camera and the total time may be based on a mode of operationof the camera system.

In some embodiments, block 704 may include determining the exposure timebased on a tunable duration that represents an acceptable amount of timeto dedicate to the autofocus procedure.

At block 706, the method 700 may include capturing, by the camerasystem, a plurality of frames for the low-light autofocus procedure. Atleast one of the extended frames may be captured by the camera systemusing the determined exposure time.

In some embodiments, at block 706, at least one of the extended framesmay be captured by the camera system using an exposure time differentthan the determined exposure time.

At block 708, the method 700 may include determining, based on theplurality of captured frames, an in-focus lens setting for a lens of thecamera system and a confidence level for the in-focus lens setting.

In some embodiments, block 708 may include applying a phase-detectionautofocus algorithm. Further, the plurality of captured frames mayinclude a first-captured frame and one or more subsequently capturedframes captured after the first-captured frame. The method 700 may alsoinclude performing facial recognition on the first-captured frame toidentify a region of interest in the first-captured frame andidentifying corresponding regions of interest in the one or moresubsequently captured frames. In addition, applying the phase-detectionautofocus algorithm may include using the phase-detection autofocusalgorithm to determine the in-focus lens setting based on thecorresponding regions of interest in the subsequently captured frames.

At block 710, the method 700 may include performing a comparison of theconfidence level to a confidence threshold.

At block 712, the method 700 may include adjusting the lens based on thecomparison.

In some embodiments, block 712 may include adjusting the lens to matchthe in-focus lens setting in response to the confidence level beinggreater than or equal to the confidence threshold.

In some embodiments, block 712 may include adjusting the lens to adefault lens setting in response to the confidence level being less thanthe confidence threshold. The default lens setting may provide a maximumfocal length for the lens. Alternatively, the default lens setting mayprovide a focal length for the lens that corresponds to a hyperfocaldistance of the camera system.

In some embodiments, the method 700 may also include capturing, by thecamera system, a plurality of additional frames. Each of the additionalframes may be captured by the camera system using a secondary exposuretime. The secondary exposure time may be shorter than the determinedexposure time. Further, the method 700 may include aligning theadditional frames such that similar objects in the additional frames arein similar pixel locations within each of the additional frames. Inaddition, the method 700 may include forming a composite image based onthe additional frames. In some embodiments, the method 700 mayadditionally include enhancing the composite image using a fast Fouriercolor constancy algorithm. Still further, the method 700 may includedetermining an optimized tone mapping for the composite image. Evenfurther, the method 700 may include modifying the composite imageaccording to the optimized tone mapping.

In some embodiments, the method 700 may include adjusting the lens to apre-autofocus setting. The pre-autofocus setting may include a middlefocal position for the lens.

In some embodiments, the camera system of method 700 may be a componentof a mobile device. As such, the method 700 may include receiving, via auser interface of the mobile device, an indication that a still image ora video image is to be captured using the mobile device. Additionally oralternatively, the method 700 may include displaying, on a userinterface of the mobile device, an indication to a user to hold themobile device still.

In some embodiments, the method 700 may include capturing, by the camerasystem and prior to determining the exposure time, a plurality ofpreview frames. The motion-blur tolerance is based on a center-weightedaverage of motion across the plurality of preview frames.

In some embodiments, the method 700 may include determining themotion-blur tolerance based on a phase-detection autofocus algorithm.

IV. CONCLUSION

The present disclosure is not to be limited in terms of the particularembodiments described in this application, which are intended asillustrations of various aspects. Many modifications and variations canbe made without departing from its scope, as will be apparent to thoseskilled in the art. Functionally equivalent methods and apparatuseswithin the scope of the disclosure, in addition to those describedherein, will be apparent to those skilled in the art from the foregoingdescriptions. Such modifications and variations are intended to fallwithin the scope of the appended claims.

The above detailed description describes various features and operationsof the disclosed systems, devices, and methods with reference to theaccompanying figures. The example embodiments described herein and inthe figures are not meant to be limiting. Other embodiments can beutilized, and other changes can be made, without departing from thescope of the subject matter presented herein. It will be readilyunderstood that the aspects of the present disclosure, as generallydescribed herein, and illustrated in the figures, can be arranged,substituted, combined, separated, and designed in a wide variety ofdifferent configurations.

With respect to any or all of the message flow diagrams, scenarios, andflow charts in the figures and as discussed herein, each step, block,operation, and/or communication can represent a processing ofinformation and/or a transmission of information in accordance withexample embodiments. Alternative embodiments are included within thescope of these example embodiments. In these alternative embodiments,for example, operations described as steps, blocks, transmissions,communications, requests, responses, and/or messages can be executed outof order from that shown or discussed, including substantiallyconcurrently or in reverse order, depending on the functionalityinvolved. Further, more or fewer blocks and/or operations can be usedwith any of the message flow diagrams, scenarios, and flow chartsdiscussed herein, and these message flow diagrams, scenarios, and flowcharts can be combined with one another, in part or in whole.

A step, block, or operation that represents a processing of informationcan correspond to circuitry that can be configured to perform thespecific logical functions of a herein-described method or technique.Alternatively or additionally, a step or block that represents aprocessing of information can correspond to a module, a segment, or aportion of program code (including related data). The program code caninclude one or more instructions executable by a processor forimplementing specific logical operations or actions in the method ortechnique. The program code and/or related data can be stored on anytype of computer-readable medium such as a storage device including RAM,a disk drive, a solid state drive, or another storage medium.

The computer-readable medium can also include non-transitory,computer-readable media such as computer-readable media that store datafor short periods of time like register memory and processor cache. Thecomputer-readable media can further include non-transitory,computer-readable media that store program code and/or data for longerperiods of time. Thus, the computer-readable media may include secondaryor persistent long term storage, like ROM, optical or magnetic disks,solid state drives, compact-disc read only memory (CD-ROM), for example.The computer-readable media can also be any other volatile ornon-volatile storage systems. A computer-readable medium can beconsidered a computer-readable storage medium, for example, or atangible storage device.

Moreover, a step, block, or operation that represents one or moreinformation transmissions can correspond to information transmissionsbetween software and/or hardware modules in the same physical device.However, other information transmissions can be between software modulesand/or hardware modules in different physical devices.

The particular arrangements shown in the figures should not be viewed aslimiting. It should be understood that other embodiments can includemore or less of each element shown in a given figure. Further, some ofthe illustrated elements can be combined or omitted. Yet further, anexample embodiment can include elements that are not illustrated in thefigures.

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purpose ofillustration and are not intended to be limiting, with the true scopebeing indicated by the following claims.

1. A method comprising: receiving an indication of a low-light conditionfor a camera system; determining an extended exposure time for alow-light autofocus procedure of the camera system; capturing, by thecamera system, an extended frame for the low-light autofocus procedure,wherein the extended frame is captured by the camera system using thedetermined extended exposure time; and determining, based on thecaptured extended frame, an in-focus lens setting for a lens of thecamera system.
 2. The method of claim 1, wherein determining an extendedexposure time for a low-light autofocus procedure of the camera systemis based on a motion-blur tolerance and the low-light condition.
 3. Themethod of claim 1, further comprising: adjusting the lens based on thedetermined in-focus lens setting.
 4. The method of claim 1, furthercomprising: determining a confidence level for the in-focus lenssetting; performing a comparison of the confidence level to a confidencethreshold; and adjusting the lens based on the comparison.
 5. The methodof claim 4, wherein adjusting the lens based on the comparisoncomprises: adjusting the lens to match the in-focus lens setting inresponse to the confidence level being greater than or equal to theconfidence threshold.
 6. The method of claim 4, wherein adjusting thelens based on the comparison comprises: adjusting the lens to a defaultlens setting in response to the confidence level being less than theconfidence threshold.
 7. The method of claim 6, wherein the default lenssetting provides a maximum focal length for the lens.
 8. The method ofclaim 6, wherein the default lens setting provides a focal length forthe lens that corresponds to a hyperfocal distance of the camera system.9. The method of claim 1, further comprising: capturing, by the camerasystem, a plurality of extended frames for the low-light autofocusprocedure, wherein determining the in-focus lens setting for the lens ofthe camera system is based upon the plurality of captured extendedframes.
 10. The method of claim 3, further comprising: capturing, by thecamera system using the adjusted lens, a second extended frame for thelow-light autofocus procedure; and determining, a second in-focus lenssetting for the lens of the camera system based upon the second extendedframe.
 11. The method of claim 10, further comprising: adjusting thelens based upon at least of one the following: the second in-focus lenssetting, a confidence level associated with the second in-focus lenssetting, and a confidence level associated with the in-focus lenssetting determined prior to the second in-focus lens setting.
 12. Themethod of claim 9, wherein at least one of the extended frames iscaptured by the camera system using an exposure time different than thedetermined extended exposure time.
 13. The method of claim 1, furthercomprising: capturing, by the camera system, a plurality of additionalframes, wherein each of the additional frames is captured by the camerasystem using a secondary exposure time, and wherein the secondaryexposure time is shorter than the determined extended exposure time;aligning the additional frames such that similar objects in theadditional frames are in similar pixel locations within each of theadditional frames; and forming a composite image based on the additionalframes.
 14. The method of claim 13, further comprising: enhancing thecomposite image using a fast Fourier color constancy algorithm;determining an optimized tone mapping for the composite image; andmodifying the composite image according to the optimized tone mapping.15. The method of claim 1, further comprising: detecting the low-lightcondition for the camera system.
 16. The method of claim 15, whereindetecting the low-light condition for the camera system comprises:comparing an ambient light intensity to a threshold low-light intensity.17. The method of claim 16, wherein the threshold low-light intensity is1.0 lux.
 18. The method of claim 1, further comprising adjusting thelens to a pre-autofocus setting, wherein the pre-autofocus settingcomprises a middle focal position for the lens.
 19. The method of claim1, wherein the camera system is a component of a mobile device.
 20. Themethod of claim 19, further comprising: receiving, via a user interfaceof the mobile device, an indication that a still image or video image isto be captured using the mobile device.
 21. The method of claim 20,further comprising: displaying, on the user interface of the mobiledevice, an indication to a user to hold the mobile device still.
 22. Themethod of claim 1, wherein determining the extended exposure timefurther comprises: determining the extended exposure time based on atotal time over which the low-light autofocus procedure is to beperformed and a number of frames to be captured during the low-lightautofocus procedure.
 23. The method of claim 22, wherein the camerasystem is part of a mobile device or a digital single-lens reflex (DSLR)camera, and wherein the total time is based on a mode of operation ofthe camera system.
 24. The method of claim 1, wherein determining, basedon the captured extended frame, the in-focus lens setting for the lensof the camera system comprises: applying a phase-detection autofocusalgorithm.
 25. The method of claim 24, further comprising: capturing aplurality of frames, wherein the plurality of captured frames includes afirst-captured frame and one or more subsequently captured framescaptured after the first-captured frame; and performing facialrecognition on the first-captured frame to identify a region of interestin the first-captured frame and identifying corresponding regions ofinterest in the one or more subsequently captured frames, whereinapplying the phase-detection autofocus algorithm comprises using thephase-detection autofocus algorithm to determine the in-focus lenssetting based on the corresponding regions of interest in thesubsequently captured frames.
 26. The method of claim 1, whereindetermining the extended exposure time further comprises: determiningthe extended exposure time based on a tunable duration that representsan acceptable amount of time to dedicate to the low-light autofocusprocedure.
 27. The method of claim 2, further comprising: capturing, bythe camera system and prior to determining the extended exposure time, aplurality of preview frames, wherein the motion-blur tolerance is basedon a center-weighted average of motion across the plurality of previewframes.
 28. The method of claim 2, further comprising: determining themotion-blur tolerance based on a phase-detection autofocus algorithm.29. A non-transitory, computer-readable medium having instructionsstored therein, wherein the instructions, when executed by a processor,perform a method comprising: receiving an indication of a low-lightcondition for a camera system; determining an extended exposure time fora low-light autofocus procedure of the camera system; causing the camerasystem to capture an extended frame for the low-light autofocusprocedure, wherein the extended frame is captured by the camera systemusing the determined extended exposure time; and determining, based onthe captured extended frame, an in-focus lens setting for a lens of thecamera system.
 30. A mobile device comprising: a camera systemcomprising: an image sensor; and a lens configured to modify light froman environment surrounding the mobile device prior to the light beingdetected by the image sensor; and a controller, wherein the controlleris configured to: receive an indication of a low-light condition for thecamera system; determine an extended exposure time for a low-lightautofocus procedure of the camera system; cause the camera system tocapture an extended frame for the low-light autofocus procedure, whereinthe extended frame is captured by the camera system using the determinedextended exposure time; and determine, based on the captured extendedframe, an in-focus lens setting for the lens.